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ABSTRACT 



A method of adapting multimedia content to a client device, 
wherein the multimedia content includes one or more items 
and the client device has capabilities and resources associ- 
ated therewith, is provided. The method includes transcod- 
ing the multimedia content into a plurality of transcoded 
content versions, wherein the plurality of transcoded content 
versions have different modalities and resolutions associated 
therewith. Next, the transcoded content versions that are not 
compatible with client device capabilities are filtered out. 
Then, at least a portion of the resources associated with the 
client device are allocated among the one or more items of 
the multimedia content. Lastly, one or more of the 
transcoded versions of the multimedia content are selected 
to generate a customized content based on allocation of the 
client device resources. 

31 Claims, 5 Drawing Sheets 
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METHODS AND APPARATUS FOR 
ADAPTING MULTIMEDIA CONTENT FOR 
CLIENT DEVICES 

CROSS REFERENCE TO RELATED 
APPLICATIONS 

The present application is related to U.S. patent applica- 
tion Ser. No. 09/413,515, entitled: "Method and Apparatus 
For Transcoding Multimedia Using Content Analysis," filed 
in the names of Li et al. on Oct. 6, 1999, which claims 
priority to U.S. provisional patent application Ser No. 
60/103,303, entitled: "Method and Apparatus For Transcod- 
ing Multimedia Using Content Analysis/' filed in the names 
of Li et al. on Oct. 6, 1998. 

FIELD OF THE INVENTION 

The present invention relates to delivery of multimedia 
content to client devices and, more particularly, to methods 
and apparatus for adapting such multimedia content for 
diverse client devices. 

BACKGROUND OF THE INVENTION 

Web documents delivered on the Internet are multimedia 
presentations that may include video, images, graphics, text 
and audio. Due to the recent rapid growth of devices that are 
connected to the Internet, there is a growing demand for 
providing universal access to such multimedia content to a 
wide variety of devices over a wide range of network 
environments. For example, personal computers on a local 
area network (LAN), personal digital assistants (PDAs) on 
dial-up modems and smart cellular phones have drastically 
different client resources in terms of, for example, screen 
size, resolution, color depth, network bandwidth and com- 
puting power. Internet users also vary in their ability to pay 
for Internet services and in the time they are ready to wait 
for a page to download. Therefore, to provide universal 
access to the Internet, multimedia delivery methods need to 
account for the composite nature of Web documents, and the 
variety of client platform capabilities, user interests, network 
constraints and authoring policies. 

In this context, video -conferencing systems have been 
proposed that adjust the bandwidth available to the client by 
selecting a suitable compression factor or codec. In these 
systems, only a single type of multimedia item (namely 
video) is considered. Also, clients that can not handle video 
are not considered. 

One option for content adaptation is to manually develop 
multiple versions of multimedia content, each suitable for a 
class of client devices. Given the variety of client devices, it 
is difficult for content publishers to anticipate and accom- 
modate the wide spectrum of client capabilities. For com- 
posite multimedia documents, such as Web pages, a number 
of systems have been proposed that employ a proxy between 
the Web server and the client. For example, various proxy 
approaches are described in: J. R. Smith, R. Mohan, and 
C-S. Li, "Transcoding Internet Content for Heterogeneous 
Client Devices," In Proc. IEEE Inter. Symp. on Circuits and 
Syst. 0SCAS), Special Session on Next Generation Internet, 
June 1998; A. Fox, S. D. Gribble, E. A. Brewer, and E. Amir, 
"Adapting to Network and Client Variability Via 
On-demand Dynamic Distillation," In ASPLOS-VII, 
Cambridge, Mass., October 1996; A. Ortega, F. Carignano, 
S. Ayer, and M. Vetterli, "Soft Caching: Web Cache Man- 
agement Techniques for Images," In IEEE Workshop on 
Multimedia Signal Processing, pg. 475-480, Princeton, N J., 
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June 1997; Intel Quick Web accessible on the Internet at 
http://www.intel.com/quickweb; Spyglass Prism accessible 
on the Internet at http;//www.spyglassxom/products/prism; 
A. Fox and E. A. Brewer, "Reducing WWW Latency and 

5 Bandwidth Requirements by Real-time Distillation," In 
Proc. Of the 5th International WWW Conference, 1996; and 
T W Bickmore and B. N. Schilit, "Digestor: Device - 
independent Access to the World Wide Web," In Proc. Of the 
6th International WWW Conference, 1997. The proxy 

10 distills, or transcodes, the content from the Web server. This 
transcoding is primarily limited to the compression of 
images, or a reduction of their size or color space. These 
systems do not consider transcoding into different modali- 
ties. The image compression and size reduction policies are 

15 static and do not dynamically account for resources on the 
client. 

SUMMARY OF THE INVENTION 

The present invention adapts multimedia content, e.g., 

20 Web documents, to optimally match the capabilities of the 
client device requesting it. Each Web document is a set of 
items, each of which is authored in a particular modality 
such as text or image. Each of these content items is then 
transcoded into multiple resolution and modality versions so 

25 that they can be rendered on different devices. For example, 
a video item is transcoded into a selected set of images so 
that it can be rendered on a device not capable of displaying 
video. Each version of a content item requires different 
resources from the client device. The invention ensures that 

30 the resource requirements for the entire document, as given 
by the sum of the resource requirements of its constituent 
items, can be met by the requesting client. The invention 
allocates the resources on the client among the items in the 
document. This resource allocation results in the selection of 

35 appropriate resolution or modality of the content items. If 
the client has limited resources, e.g., such as a PDA or pager, 
some of the content items may not get any resources 
assigned and thus not be delivered to the client. 

40 In an embodiment of the invention, as will be explained, 
three technologies are employed to provide such multimedia 
content adaptation: (i) a progressive data representation 
scheme referred to as the InfoPyramid as described in C-S. 
Li, R. Mohan and J. R. Smith, "Multimedia Content 

45 Description in the InfoPyramid," Proc. ICASF98, Special 
Session on Signal Processing in Modern Multimedia 
Standards, Seattle, Wash., May 1998, the disclosure of 
which is incorporated herein by reference; (ii) a set of 
transcoding modules for converting modality or resolution; 

50 and (iii) an adaptation process that selects the best repre- 
sentation to meet the client capabilities while delivering the 
most value to the client. 

The present invention provides many advantages over 
prior solutions. For example, content is dynamically adapted 

55 to the client device allowing a wider variety of multimedia 
content and of client devices to be properly supported. Also, 
in accordance with the invention, a content author has 
control over the adaptation process. The content author can 
edit and replace the transcoded versions of content items 

60 generated by the automated transcoding systems. This con- 
trol of the customization overcomes problems of publisher 
control and copyright issues faced by transcoding proxies. 

Further, the invention permits content to be authored in 
XML (Extensible Markup Language, as is known in the art), 

65 allowing the author to provide more information to the 
transcoding and adaptation systems than can be deduced 
from an HTML (hyper text markup language) page. One 
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benefit of the server-based system of the invention is that memory devices (e.g., ROM, fixed or removable memory) 
due to the guidance provided by the author, a significantly and, when ready to be utilized, loaded in part or in whole 
greater level of customization can be performed than is (e.g., into RAM) and executed by a CPU. 
possible in previous transcoding proxies. Still further, the Referring to FIG. 1, an example of a multimedia docu- 
invention permits the transcoded versions of the content 5 ment is shown. As is well known, a Web user (client) makes 
items to be generated prior to any requests. Thus, the a request from his client device for some particular Web 
invention can handle media items such as video and audio documents) or other multimedia content which may contain 
which are difficult to handle in conventional proxies. This video, images, graphics, text, audio. The request is sent to a 
off-line transcoding also leads to lower response latencies Web server which satisfies the request by returning the 
than proxies. Also, the server shares the benefit of transcod- 10 requested document to the client. As shown, a multimedia 
ing proxies in speeding content delivery as the customized document W, denoted as reference numeral 100, is corn- 
content is often much smaller than the original content. posed of a number of content items A t -, denoted as reference 
These and other objects, features and advantages of the numeral 120, where i=l, 2, . . . , n. Thus, the multimedia 
present invention will become apparent from the following document may be represented as W-{ A,}. Each content item 
detailed description of illustrative embodiments thereof, 15 A,-, can be an atomic unit of media such as, for example, an 
which is to be read in connection with the accompanying image. A content item A,- can also be composed of other 
drawings. items. For example, a document can have a number of 

stories as content items, and each story item may be com- 

BRIEF DESCRIPTION OF THE DRAWINGS p0S ed of image items, text items, etc. 

FIG. 1 is a diagram illustrating an example of a multi- 20 Referring to FIG. 2, a diagram illustrating an example of 

media document; transcoding a content item into an InfoPyramid according to 

FIG. 2 is a diagram illustrating transcoding a content item me invention is shown. Particularly, a content item 120 is 

into an InfoPyramid- transcoded, in process 250, into an InfoPyramid 280. For 

^„ «. „ . 4 . n t , c , each original content item A., an InfoPyramid represented as 

FIG, 3 is a flow chart illustrating a method or adapting a 25 w rw i • n • * j u * j- a • * • 

, i- * j ■ j- i H°{M lV },i=0, m„ is computed by transcoding A.- into j 

document to a client device according to an exemplary * > f \ fV A j iv V - - 

, , f4 , . versions with dine rent resolutions and modalities. The ongi- 

embodiment of the invention; ^ yejsiQD {& denQted fls M ^ A Qu{{ vtT&ioR ^ which 

FIG. 4 is a flow chart illustrating an adaptation process corresponds to an item being deleted from the delivered 

according to an exemplary embodiment of the invention; content is denoted as: 

FIG. 5 is a flow chart illustrating a method of obtaining an 30 

adaptation process according to an exemplary embodiment Mimj^- 0) 

of the invention; ^ , 

- . _ i.-ii... a. £ * j* The transcoding process 250 can convert the input modal- 

FIG. 6 is a flow chart illustrating a method of selecting . . . , , L . . A 1% u 

* , a * i r i j j * j ity associated with the content item to a new modality such 

versions of content items for a customized document accord- J e . . , , AUq 

. ( , lt . 4 . 35 as, for example, text to audio, or audio to text. Alternatively, 

ine to an exemplary embodiment oi the invention; . ' jf . , , 

B * * ' the transcoding process can summarize, compress, or elabo- 

FIG. 7 is a block diagram illustrating deployment of an ^ Qn ^ CQntent (he giveQ modality of the inpm 

adaptation process m the Internet according to an exemplary daU SUCQ ^ for example> by image compression or text 

embodiment of the invention; summarization. Further, it is to be appreciated that transcod- 

FIG. 8 is a block diagram illustrating an exemplary 40 i n g processes may alternatively be performed in accordance 

hardware architecture of a server or computing device for tne techniques described in U.S. patent application Sen 

implementing the invention; and n 0 09/413,515, entitled: "Method and Apparatus For 

FIG. 9 is a graphical representation illustrating a resource Transcoding Multimedia Using Content Analysis," filed in 

allocation technique according to an exemplary embodiment the names of Li et al. on Oct. 6, 1999, which claims priority 

of the invention. 45 to U.S. provisional patent application Ser, No. 60/103,303, 

„ __ VT „ entitled: "Method and Apparatus For Transcoding Multime- 

DETAILED DESCRIPTION OF PREFERRED dia Using Contcnt j^^," filed in the names of Li et al. 

EMBODIMENTS Qn Qct ^ 199g> the disclosure of which is incorporated 

It is to be appreciated that the term "processor" as used herein by reference, 
herein is intended to include any processing device, such as, 50 The InfoPyramid 280 can be viewed as a generalization of 
for example, one that includes a CPU (central processing multi-resolution representations, e.g., Gaussian image 
unit). The term "memory" as used herein is intended to pyramids, to multimedia. The InfoPyramid provides a multi- 
include memory associated with a processor or CPU, such modal, multi-resolution representation hierarchy for 
as, for example, RAM, ROM, a fixed memory device (e.g., multimedia, for example, as described in the above- 
hard drive), a removable memory device (e.g., diskette), etc. 55 referenced C-S. Li, R. Mohan and J. R. Smith, "Multimedia 
In addition, the term "input/output devices" or "I/O devices" Content Description in the InfoPyramid," Proc. ICASP'98, 
as used herein is intended to include, for example, one or Special Session on Signal Processing in Modem Multimedia 
more input devices, e.g., keyboard, for making requests for Standards, Seattle, Wash., May 1998. The InfoPyramid 280 
multimedia content and inputting data to the processing unit, is a data structure in which the multiple representations of a 
and/or one or more output devices, e.g., CRT display and/or 60 multimedia item can be organized into a pyramid-like 
printer, for presenting requested multimedia content and structure, as illustrated in FIG. 2. The cells of the pyramid 
providing results associated with the processing unit. It is correspond to different representations of the objects using 
also to be understood that various elements associated with different modalities such as, for example, video 281, image 
a processor may be shared by other processors. Accordingly, 282, audio 283 and text 284, and fidelities such as in the 
software components including instructions or code for 65 range of full-resolution (bottom of pyramid) to low- 
performing the methodologies of the invention, as described resolution (top of pyramid). A specific version of a multi- 
herein, may be stored in one or more of the associated media item can be referred to by one of the cells. The 
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InfoPyramid may include procedures and rules for translat- transcoded (250) into multiple modality and fidelity versions 

ing and summarizing (transcoding) between modalities and to generate a set 340 of InfoPyramids 280. 

resolutions. The InfoPyramid may also contain meta-data A client device is characterized by its profile 310 which 

for each constituent version such as, for example, size, color, li sts lhe capabilities 320 and resources 330 of the device, 

bandwidth requirements, publisher preferences, etc. 5 T° c tv P es of clicnt devices that can access the Internet are 

The transcoding process 250 populates the InfoPyramid ra P idl y expanding beyond the typical workstation on a LAN 
280 structure with multi-resolution, multi-modal versions of for w ^ ch most multimedia Internet content is authored 
the content item, as explained herein. The transcoding A client may have a different number of capabilities 320 
process 250 can be performed on a version by following the « d s « * l ° be appreciated that term 
f j * *u • cm *> f i a * j u capability" is the ability for a chent device to handle a 
transcoding paths in FIG 2, for example, as denoted by 10 Ocular media type. Examples of capabilities that maybe 
reference numerals 285 through 290 By following the £ ified delude: ® screen color; (ii) audio capability; (iii) 
horizontal paths, e.g., paths 285 and 286, a version can be yideo capability; (iv) image display ca p a bility; (v) software 
translated to another version in a new modality. For such ^ decompression; and (vi) hardware such as a 
example, text can be converted to audio in path 286. smmd card ^ to aame a few So> by specifying the capabilities 
Similarly, video can be converted to images in path 285. By 15 ma t a client device has, it is implied that capabilities not 
following the vertical paths, e.g., paths 287, 288, 289 and specified do not exist on the client device. For example, a 
290, a version can be changed into versions at different handheld personal computer (HPC) may not be capable of 
fidelities. For example, text passages can be summarized in displaying video and a PDA may not be capable of display- 
path 288, video can be compressed in path 287 and/or ing color images. The "resources" of a client can be divided 
images can be compressed in path 289. Furthermore, the 20 up among several items on a Web page. Examples of 
image can be compressed to a greater degree in path 290. resources are: (i) screen size including width, height and 
Such transcoding processes can be done in a "lazy" fashion, area; (ii) bandwidth, for example, network bandwidth; (iii) 
i.e., the actual transcoding need not be performed, but rather, memory and other client storage capacity; (iv) wait time 
only a place holder for the transcoded version can be created (including preferred and limits) as determined by the prod- 
and the meta-data filled in. The actual version is generated 25 uct of the bandwidth and the time a client is ready, willing 
only if required. or able to wait to receive the complete document; (v) 

Some exemplary transcoding processes 250 are listed payment ability (including preferred and limits), that is, 

below by modality of the version and the conversions money client is ready, willing or able to pay for the 

performed to generate versions at different resolutions and document, to name a few. 

modalities: 30 The client device profile can be obtained by a number of 

(i) images mechanisms. For example, the User-Agent fields in the 
resolution— spatial size reduction, color depth reduction, HTTP (hyper text transport protocol) request header contain 

lossy/lossless compression information about the browser and often the operating 

modality-related text, embedded text, semantic labels; svstem - i ^ so ' Windows-CE devices specify the screen size, 

(ii) video 35 co ^ or depth and processor. Standardization efforts are under 

. . , 4 , 14 . , , way to allow these request fields to contain more informa- 

resolution — spatial and temporal resolution, lossy/ . J . - v . ^ . „ 

lossless com ression about the client device. Further, many sites require users 

oss ess compression ^ log-in or place cookies at the user location allowing client 

modality— key-frame images, audio track, closed caption; capabilit i es to be retrieved from stored profiles. The client 

(iii) text 

40 may also specify their capabilities explicitly, for example, 

resolution — key-terms, text summarization, document through forms or applets. 

headings A content adaptation process 350 uses the client profile 

modality— speech synthesis, language translation; 310 to select from among the InfoPyramids 280 the versions 

(iv) audio 374 that best satisfy the particular client profile. These 
resolution — bit-rate reduction, sampling rate change, ste- 45 selected versions are rendered into a document 370 which is 

reo to mono an adaptation (i.e., customization) of the original multimedia 

modality — speech recognition. document 100. The client device receives the customized 

That is, such known transcoding operations are capable of document 370. 

transcoding multimedia content in a variety of ways. Some Referring to FIG. 4, a flow chart illustrating an adaptation 

illustrative examples of the exemplary conversions men- 50 process 350 according to the invention is shown. First, in 

tioned above include; (i) compressing data with or without step 420, the versions from the InfoPyramids 280 that do not 

information loss; (ii) reducing the size or resolution of the meet client capabilities 320 are filtered out. For example, all 

data such as by spatial reduction; (iii) reducing the color versions that are in video modality can be filtered out if the 

content of the data such as reducing the number of colors; device is not capable of displaying video. Similarly, all 

(iv) summarizing content such as text passages and video 55 versions that are color images can be filtered out for a device 

sequences; (v) reducing or expanding the amount of data or that has only a black and white or gray-level screen, 

altering the fidelity of the information; (vi) translating the Next, in step 450, the client resources are allocated 

data to different modalities such as text to speech, video to between the InfoPyramids 280 and the content items. The 

images, and speech to text; (vii) translating text to a different resources can be allocated by using any allocation resource 

language such as English to Japanese; and (viii) removing, 60 policy. For example, the resources can be distributed in 

substituting, blocking, emphasizing or de-emphasizing indi- proportion to the resources used by the original versions. A 

vidual multimedia objects, modal elements, or other content. method for allocating resources may be based on 

Referring to FIG. 3, a flow chart of a method 300 of optimization, as will be explained in the context of FIG. 5. 

adapting a multimedia document 100 to a client device Then in step 470, the resources allocated for each InfoPy- 

according to the invention is shown. A client device is 65 ramid are used to select the appropriate version 374, which 

characterized by a client profile 310. As shown in FIG. 2, the has not been previously filtered out in step 420, to render the 

content items 120 of a multimedia document 100 are customized multimedia document 370. 
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Referring to FIG. 5, a flow chart illustrating a method 450 
of allocating client resources using optimization according maxj v p \ v & 

to the invention is shown. First, in step 520, each content , ]L Vi J such *"* L Ri * Hdim 

item is assigned a priority. The author of the Web document 
may have a mental priority ordering of the items in the 5 

document. The priorities may be assigned by the author of where V t -e{V(M () -)} and R^{R (M (V )} are the values and the 
the page. Many Internet applications, such as search resources used by the \ th item M, of the multimedia docu- 
engines, customized news sites, etc, generate documents ment. While V, and R, are discrete, we will first consider 
dynamically in response to a user request. In these them to be continuous, and then deal with the discrete case, 
applications, there is often a priority implicitly assigned to 10 Ku en t is the maximum resource available at the client, 
the items. For example, in image search engines, the match Let the value V, be some function of the resource R„ i.e. 
scores of the returned images serve as priorities. Priorities ^UU -We convert the above constrained op Umiza ion 
can be assigned based on match scores for various dynami- ^ cm Ration (5)) to an unconstrained optimization 
cally generat^ Stc^ 
priorities can be assigned on attributes of content » For 15 0 £ timizati » John wn & Sons> 1987 , Using the 
example, titles can be given a higher priority than headings jj^rangiaa technique, equation (5) may be expressed as: 
which can be given a higher priority than paragraphs. 

Content analysis or heuristics give hints to semantics or . 
purpose of various content items. Such content analysis, for w a) = J V v, +\\ R ctUnt - £ R, | with A * 0. 

example, is described in S Paek and J. R. Smith, "Detecting 20 L . v / J J 

Image Purpose in World Wide Web Documents/' In Symp. 

On Electronic Imaging: Science and Technology- ^^tfR isanoptima ^^ 

Document Recognition, San Jose, Calif., January 1998, the a x ^ ^ VL(R ,XW). Thus, the candidate solutions for 
disclosure of which is incorporated herein by reference, ^ maxirmim are given by: 
while heuristics are, for example, described in J. R. Smith, 25 

R. Mohan and C-S. Li, "Content-based Transcoding of QL dL dL dL (7) 

Images in the Internet," In IEEE Inter. Conf. On Image IF = IF = '" IF ^"'Ix = d ' 

Processing (ICIP-98), Chicago, 111., October 1998, the dis- 12. 
closure of which is incorporated herein by reference. For 

example, one can categorize images as pictures, graphics, 30 Given that the items and their values are independent of each 
advertisements, etc. These categories can be used to assign other, we get: 
priorities. Also, priority may be assigned to items based on 

subjective or objective importance. In any case, let P ( - be the _ <JM R >) _^ ( 8 ) 

priority assigned to item i. dR < dR < 

Next, in step 540, each version of a content item in an 35 
InfoPyramid is assigned a value. A value is a subjective j^f^ the candidate solutions to equation (5) are given 
measure of fidelity and may be represented as: ^y: 

perceived value of transcoded version M ts (2) , , . . , „ ,„ . iQ , 

V(M :: ) = — — 7 , „ - An dflWl) _ dflWl) dfiWi) _ . (9) 

perceived value of original M\q 40 ~~dF\ — ~ dR% "' dk"~ "' ~ 

where; 

v e p. i], ( 3 ) First we consider analytic functions. Content value, as an 

for original item alternative distortion, makes it possible for authors or users 

,. . . , - . 45 to specify value judgements about various transcoded ver- 

when the item is excluded from M im . • r c iL * * tt m vi- 

sions of the content. However, manually assignmg the 

values is not a practical proposition in most scenarios. To 

„ „ .. A mitigate this problem, we introduce functional mappings 

V provides a measure of fidelity that is apphcab e to between content value and resource utilization, 

transcodings of media at multiple resolutions and multiple Ut us assume a fonction ^ v-fXVJ. Note that /, and 

modalities. This also allows companson of content items therefore, the solution, is dependent on the choice of units 

that are in different media types. The value V can either be for R If f is con cave, equation (9) gives us the optimal 

assigned by the author for each transcoding, or a functional solution. We will first however consider the case when f is 

relation between V and R (i.e., the resource utilized) can be no t concave followed by the case when it is concave, 

defined. In the special case where a distortion D of all the Non-concave case: We will limit the discussion to the case 

versions can be measured, and the distortion for the null 55 when f is either linear or convex. Let us assume that the 

version is assumed to be infinite, V can be computed as: va i ue 0 f an item is linearly proportional to the resource that 

it utilizes, i.e., V=c ( R ( . From the definition of V we have that 

v _ l ( 4 ) f t (R^)=Q when item i is absent from the delivered document, 

1 +■ D ' i.e., R ( =0 and f t (R^l for the original version of item i, i.e., 

60 R,=R ; mar . Thus: 



(10) 



The prioritized value of InfoPyramid M t - is defined as 1 

V^aPfVy. Next, in step 560, the client resources 320 are c » = ^r- 
allocated by maximizing, or at least substantially 

maximizing, the sum of prioritized values of the InfoPyra- 65 

mids of all content items such that the client resources are We term c t to be the resource utilization factor RUF because 

not exceeded, i.e.: it measures how well the item i utilizes its resources to 
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deliver value. It is evident that a greedy algorithm that 

allocates resources to items in the order of their RUFs gives y ^ (14) 

the optimal resource allocation: i 

1. Store items in order of decreasing RUF, c. 

2. Starting with the item with the largest RUF, allocate the 5 ^ close to but less than R p oin|s outside fte concave 

re^™a^TTeted atea emCaDUSeUn hU " m U ° l * lhe SOluti ° n F ° r eXample ' * tCXt 

" ' , transcript of video may take more screen space but have less 

Sinnlarly, the optimal resource allocation for any convex va , ue> ^ it ^ out of ^ somon space. 

function fis also a greedy algorithm. 10 Recall that P, is the priority assigned to item i by .he 

Concave case: Ut us consider the concave function V,-c,. aulhof of ^ lication . Wc defiac thc rioritiz6d 

In (HR,). We have defined f on 1 + R, to avo.d negative ^ of Uem . ^ y P _ p y llle , h n(JW to find . 

For simplicity, we assume that R»l for most versions, and 
that R«l is equivalent to the item being deleted, giving: 



15 



ln(/? t ) (11) 



t{£ such z s /?c *< n " 



(15) 



Using this formulation, the following resource allocation 

We now get a resource utilization factor RUF of: strategies are similar to those described above but with 

20 prioritized values replacing RUFs: 

c . = 1 ( 12 > 1. If f is linear or convex, the resources are assigned in a 

intfff 11 *) greedy manner in order of the prioritized values of the 

items. 

Using equation (9), we see that the resources are distributed 25 2. If f is logarithmic, the resources are assigned in 
among the items in proportion to their RUFs. Since, V-c,. In proportion to the prioritized values of the items. In 
(R^) is concave (and the constraint is linear) this solution is general, when / is concave, equation (15) can be solved 
optimal. In a similar manner, equation (9) yields the optimal b y tne technique described above with respect to ana- 
solution for all other concave functions. functions. 

Now considering discrete values, since each item is 30 ™«>" I * «* { \%^ vers "? resource ^ { 

transcoded into a limited number of versions, we may have < FIG ; *) 15 "P^f Wlt u h V < = P ' V < 0D ^ y~ axis - ™? 

no version that uses exactly the same resource as computed remainder of the algorithm is as described above with 

in the optimization process described above. To account for respect to arbitrary functions. 

the discrete values, we use the following algorithm: ° ne common transcoding practice is to scale the size of 

35 all images by a constant factor: 0.75, 0.25, etc. One can 

1. For each item i, let R/ be the resource selected by the ai ^ uc that in ^ original Web document, the larger images 
optimization process. Select version j such that R jy <R/ W ere more important as more resources were given to them, 
and R/-R tf is minimum. \y e can model these intrinsic properties as proportional to 

2. Order the items in order of decreasing RUFs. Starting ln(image size). Then, this ad-hoc policy of constant scaling 
from the item with the highest RUF, while there are any 40 is equivalent to allocating the client resources of screen area 
resources left, assign to each item the version with the in proportion to the prioritized values of the images, and is 
next highest value. optimal with the assumption that image value is a logarith- 

It is to be appreciated that step 2 needs to be performed only mic function of its size. 

once. Each content item i can be a composite item, i.e., a 

Next we consider arbitrary functions. When the values V. 45 hierarchy of other content items. To account for composite 

are assigned, for example by the author, we may not have an items > we allocate resources using Vff&d where / is a 

analytical function j^R,) and thus will not be able to concave analytic function and the items 1 under consider- 

calculate: ' ' al * on mav be composite. The optimal resource R/ thus 

allocated to each composite item i is in turn used as the 

dftft) (13) 50 resource constraint for its constituent items. We then allocate 

■ this resource R/ among the children of the composite item 

i. This resource allocation is repeated until the items being 
considered are atomic (e.g., an individual media item such 

For this case, we adapt a technique described in Y. Shoham as an image). When we have priorities assigned to items, we 

and A. Gersho, "Efficient Bit Allocation for an Arbitrary Set 55 similarly modify the resource allocation strategy as 

of Quantizers," IEEE Trans. On Acoust., Speech, Signal described above. 

Proc, 36(9): 1445-1 453, September 1998, the disclosure of For a composite item, the number of its different versions 

which is incorporated herein by reference. For each Infopy- is combinatorial in the number of its children items. Thus, it 

ramid M, of each item i, we plot the value \ if versus the is not always practical to manually assign values to each 

resource utilized R ( y of each version M,y, as illustrated in 60 version of a composite item. 

FIG. 9. The optimal version M/ given by sweeping a line As previously mentioned, a client may have a different 

with slope X from the top-left to the bottom-right until it number of capabilities and resources. Before we start the 

meets the concave hull of these points. As shown by resource allocation process, we remove from consideration 

equation (9) and as known in the above -referenced Shoham all the versions of items that a client is not capable of 

article, the optimal solution is given by the same slope X for 65 handling. Let there be r different resources R fc c / X > m that we 

all the different items i. Then, a binary search for X is have to consider. Then, the resource allocation problem can 

performed such that: be stated as: 
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maxf_ ) „ ^ (16) 

/ |2j V; J such ^ Zj ^ s for 311 1 s * 5 r * 

We first allocate each resource K k cHent separately. Let M * be 
the version of item M ( - that is selected for resource k. We 
select only one version of an item to be delivered. We 
achieve this by the following algorithm: 

1. For each item i find the set of resources R*(M,-*)} 
used by each of its versions selected for each of the 
resources. 

2. For each item i find the version M iy that has the highest 
assigned value among all the versions and such that for 
all k, R*(M iy )^R* (M £ *)eR t , If no values have been 
assigned, since we define V^j^R/), the value of a 
version may be different for different resources. In that 
case, we choose a resource as the dominant resource, 
and use the values of the versions for that resource. 

The above algorithm is guaranteed to select one version 
for each item without breaking the constraints of any of the 
r resources. However, the version so selected may not 
always be optimal. To find the optimal set, a search (e.g., 
combinatorial) may be required. 

For finding the optimal content adaptation schemes we 
assumed that the content items on a Web page are indepen- 
dent of each other. This assumption may not hold in all 
cases. For example, for a news story, if the text to the story 
has to be discarded due to space limitations, then delivering 
the pictures for the story may not be very useful. One 
solution is to use composite items, as described above. We 
consider dependent items as composite items and allocate 
resources first to the composite item. This resource can then 
be allocated among the components of the composite item in 
an all-or-none manner. Another solution is to extend rate- 
distortion techniques for handling dependent blocks such as, 
for example, is described in A. Ortega and K. Ramchandran, 
"Rate-Distortion Methods for Image and Video 
Compression," IEEE Signal Processing Magazine, Novem- 
ber 1998, to the value -resource framework. 

Advantageously, given that client resource allocation is 
expressed in the form of the above optimization problem, 
any of the above described techniques may be employed. It 
is to be appreciated that the above techniques are also 
described in R. Mohan, J. R. Smith and C-S. Li, "Adapting 
Multimedia Internet Content for Universal access/' IEEE 
Transactions on Multimedia, Vol. 1, No. 1, March 1999, the 
disclosure of which is incorporated herein by reference. It is 
further to be appreciated that techniques known to those 
ordinarily skilled in the art for computing the solution to this 
type of optimization problem may alternately be employed. 
Whichever technique is used to find the optimum value, the 
solution gives the amount of resource R ( allocated to the 
InfoPyramid M t - for content item A,-. Then, the versions are 
selected based on the resources that have been allocated 
(step 470). 

Referring to FIG. 6, a flow chart illustrating a method 470 
of selecting versions based on the resources allocated (step 
450) according to the invention is shown. For each InfoPy- 
ramid M, for content item A,-, let R, 1 be the resource allocated 
by step 450. In step 640, for each InfoPyramid, select 
version j such that R iy =R/ and R^-R^ is a minimum. In step 
680, the InfoPyramids are sorted in the order of decreasing 
prioritized values. Starting from the item with the highest 
prioritized value, while there are any resources left, select 
from the InfoPyramid of each item the version with the next 
highest value. Step 680 needs to be performed only once. 
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These selected versions 374 are rendered as the multimedia 
document 370 customized for the client device. 

It is to be appreciated that in an alternative embodiment, 
the fidelity measure values, priorities, and the Infopyramids 
s may be made part of the original multimedia document. If 
this is the case, then only the selection process (FIG. 6) need 
be performed to generate the customized multimedia docu- 
ment 370. 

Referring to FIG. 7, a block diagram illustrating an 
example of deployment of a content adaptation process 300 
according to the invention in a client-server environment is 
shown. It is to be appreciated that the content adaptation 
process can be deployed on the server-side, the client -side, 
or in the network connecting the server and client. For 
example, the content adaptation process 300 may be imple- 

15 mented in a server 701 or a computing device 704 on the 
server-side. Further, the process 300 may be implemented in 
a proxy server 705 in the network. Still further, the process 
300 may be implemented in a client device 703 or a 
computing device 706 on the client-side. 

20 If the content adaptation process 300 is deployed at the 
proxy server 705, the proxy handles the requests from the 
client devices 703 for multimedia content (e.g., documents). 
The proxy retrieves the requested information from servers) 
701. Then, the process 300 running on the proxy adapts the 

25 retrieved content to the profile 310 of the requesting client 
device 703. The proxy 705 then delivers the customized 
document 370 to the client device 703. If the content 
adaptation process 300 is deployed at the server 701, the 
process 300 has direct access to the content at the server 701 

30 and can store the InfoPyramids 280 on the server 701. If the 
content adaptation process 300 is deployed at the client-side 
computing device 706, the process 300 can adapt documents 
retrieved by the clients 703 from the server(s) 701. 
Alternatively, the process 300 may be distributed among two 

35 or more locations. 

Referring to FIG. 8, a block diagram is shown of an 
exemplary hardware implementation for any of the com- 
puter systems used to implement the methodologies of the 
invention, e.g., server(s) 701, client devices 703, computing 

40 device 704, proxy server 705, and computing device 706. 
Each computer system includes a processor 810 coupled to 
a memory 820 and I/O device(s) 830. The processor 810 
performs the functions associated with the methodologies 
described herein. The memory 820 is used by the processor 

45 for performing such functions and for storing results of the 
processes. The I/O device(s) may include one or more data 
input devices (e.g., keyboard, etc) for inputting data (e.g., 
requests for multimedia content), and/or one or more data 
output devices (e.g., display) for presenting results associ- 

50 ated with the functions performed in accordance with the 
methodologies described herein (e.g., displaying customized 
documents). Also, it is to be understood that more than one 
processor may be employed within any one computer sys- 
tem and various processors may share responsibility for 

55 executing processes associated with the invention. While the 
above embodiments illustrate a processor-based implemen- 
tation of the invention, one of ordinary skill in the art will 
realize that other implementations may be used in whole or 
in part, e.g., application specific integrated circuits, special 

60 purpose circuitry, etc. 

Although illustrative embodiments of the present inven- 
tion have been described herein with reference to the accom- 
panying drawings, it is to be understood that the invention 
is not limited to those precise embodiments, and that various 

65 other changes and modifications may be affected therein by 
one skilled in the art without departing from the scope or 
spirit of the invention. 
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What is claimed is: 

1. A method of adapting multimedia content to a client 
device, the multimedia content including one or more items 
and the client device having capabilities and resources 
associated therewith, the method comprising the steps of: 

transcoding the multimedia content into a plurality of 
transcoded content versions, the plurality of transcoded 
content versions having different modalities and reso- 
lutions associated therewith; 

filtering out the transcoded content versions that are not 
compatible with client device capabilities; 

allocating at least a portion of the resources associated 
with the client device among the one or more items of 
the multimedia content; and 

selecting one or more of the transcoded versions of the 
multimedia content to generate a customized content 
based on allocation of the client device resources. 

2. The method of claim 1, wherein the multimedia content 
includes at least one World Wide Web document. 

3. The method of claim 1, wherein the multimedia content 
includes at least one of a video item, an image item, a 
graphics item, a text item, and an audio item. 

4. The method of claim 1, wherein the transcoding step 
further includes the steps of: 

separating the multimedia content into individual content 
items; and 

converting the individual content items to different reso- 
lutions and modalities. 

5. The method of claim 1, wherein the capabilities asso- 
ciated with the client device include at least one of a color 
capability, an audio capability, a video capability, an image 
capability, and a text capability. 

6. The method of claim 1, wherein the capabilities asso- 
ciated with the client device includes a decompression 
capability. 

7. The method of claim 1, wherein the resources associ- 
ated with the client device include at least one of a screen 
size, a bandwidth, a storage capacity, a content wait time 
ability, and a payment ability. 

8. The method of claim 1, wherein the transcoding step 
includes separating the multimedia content into individual 
content items. 

9. The method of claim 8, wherein the allocating step 
further includes assigning a priority to a content item. 

10. The method of claim 9, wherein the priority is 
assigned by an author of the multimedia content. 

11. The method of claim 9, wherein the priority is 
assigned by an application associated with the multimedia 
content. 

12. The method of claim 9, wherein the priority is 
assigned in response to a user request. 

13. The method of claim 9, wherein the priority is 
assigned in response to a match score. 

14. The method of claim 9, wherein the priority is 
assigned as a function of the importance attributed to a 
content item. 

15. The method of claim 9, wherein the allocating step 
further includes assigning a fidelity measure value to each 
content version. 

16. The method of claim 15, wherein the allocating step 
further includes computing a product of the priority and the 
fidelity value for each content version. 

17. The method of claim 15, further including the step of 
including the assigned fidelity measure value in the multi- 
media content. 

18. The method of claim 9, further including the step of 
including the assigned priority in the multimedia content. 
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19. The method of claim 1, wherein the transcoded 
versions selected satisfy at least a portion of the resource 
allocations. 

20. The method of claim 1, further including the step of 
S organizing at least one of the one or more items and at least 

a portion of the transcoded content versions associated 
therewith in a data structure. 

21. The method of claim 20, further including the step of 
including the data structure in the multimedia content. 

10 22. The method of claim 1, wherein the resources allo- 
cated to at least a portion of the transcoded versions of the 
content items are a function of priorities. 

23. A method of adapting multimedia content to a client 
device, the multimedia content including one or more items 

15 and the client device having capabilities and resources 
associated therewith, the method comprising the steps of: 
transcoding the multimedia content into a plurality of 
transcoded content versions, the plurality of transcoded 
content versions having different modalities and reso- 
20 lutions associated therewith; 

filtering out the transcoded content versions that are not 

compatible with client device capabilities; 
allocating at least a portion of the resources associated 
with the client device among the one or more items of 
25 the multimedia content; and 

selecting one or more of the transcoded versions of the 
multimedia content to generate a customized content 
based on allocation of the client device resources; 
wherein the transcoding step includes separating the mul- 
timedia content into individual content items, and 
wherein the allocating step further includes assigning a 
priority to a content item, assigning a fidelity measure 
value to each content version, computing a product of 
35 the priority and the fidelity value for each content 
version, and at least substantially maximizing a sum of 
the products for the content versions such that the 
resources associated with the client device are not 
exceeded. 

24. The method of claim 23, wherein the maximizing step 
results in determining the portion of the resources associated 
with the client device which are allocated to each content 
item. 

25. A method of adapting multimedia content to a client 
device, the multimedia content including one or more items 
and the client device having capabilities and resources 
associated therewith, the method comprising the steps of: 

transcoding the multimedia content into a plurality of 
transcoded content versions, the plurality of transcoded 
50 content versions having different modalities and reso- 
lutions associated therewith; 
filtering out the transcoded content versions that are not 

compatible with client device capabilities; 
allocating at least a portion of the resources associated 
55 with the client device among the one or more items of 
the multimedia content; and 
selecting one or more of the transcoded versions of the 
multimedia content to generate a customized content 
based on allocation of the client device resources; 
60 wherein the resources allocated to at least a portion of the 
transcoded versions of the content items are a linear 
scaling function of the resources that would otherwise 
be allocated to the items. 

26. Apparatus for adapting multimedia content to a client 
65 device, the multimedia content including one or more items 

and the client device having capabilities and resources 
associated therewith, the apparatus comprising: 
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at least one processor operable to transcode the multime- 
dia content into a plurality of transcoded content 
versions, the plurality of transcoded content versions 
having different modalities and resolutions associated 
therewith, to filter out the transcoded content versions 
that are not compatible with client device capabilities, 
to allocate at least a portion of the resources associated 
with the client device among the one or more items of 
the multimedia content, and to select one or more of the 
transcoded versions of the multimedia content to gen- 
erate a customized content based on allocation of the 
client device resources; and 

memory, coupled to the at least one processor, for storing 
at least a portion of results associated with one or more 
of the operations performed in accordance with the at 
least one processor. 

27. The apparatus of claim 26, wherein the multimedia 
content includes at least one World Wide Web document. 

28. The apparatus of claim 26, wherein the multimedia 
content includes at least one of a video item, an image item, 
a graphics item, a text item, and an audio item. 

29. The apparatus of claim 26, wherein the capabilities 
associated with the client device include at least one of a 
color capability, an audio capability, a video capability, an 
image capability, and a text capability. 
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30. The apparatus of claim 26, wherein the resources 
associated with the client device include at least one of a 
screen size, a bandwidth, a storage capacity, a content wait 
time ability, and a payment ability. 

31. An article of manufacture for adapting multimedia 
content to a client device, the multimedia content including 
one or more items and the client device having capabilities 
and resources associated therewith, comprising a machine 
readable medium containing one or more programs which 
when executed implement the steps of: 

transcoding the multimedia content into a plurality of 
transcoded content versions, the plurality of transcoded 
content versions having different modalities and reso- 
lutions associated therewith; 

filtering out the transcoded content versions that are not 
compatible with client device capabilities; 

allocating at least a portion of the resources associated 
with the client device among the one or more items of 
the multimedia content; and 

selecting one or more of the transcoded versions of the 
multimedia content to generate a customized content 
based on allocation of the client device resources. 
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